Testing real-time multi input-output systems

In formal testing, the assumption of input enabling is typically made. This assumption requires all inputs to be enabled anytime. In addition, the useful concept of quiescence is sometimes applied. Briefly, a system is in a quiescent state when it cannot produce outputs. In this paper, we relax the input enabling assumption, and allow some input sets to be enabled while others remain disabled. Moreover, we also relax the general bound M used in timed systems to detect quiescence, and allow different bounds for different sets of outputs. By considering the tioco-M theory, an enriched theory for timed testing with repetitive quiescence, and allowing the partition of input sets and output sets, we introduce the mtioco^M relation. A test derivation procedure which is nondeterministic and parameterized is further developed, and shown to be sound and complete wrt mtioco^

Testing multi input-output real-time systems (Extended version)

In formal testing, the assumption of input enabling is typically made. This assumption requires all inputs to be enabled anytime. In addition, the useful concept of quiescence is sometimes applied. Briefly, a system is in a quiescent state when it cannot produce outputs. In this paper, we relax the input enabling assumption, and allow some input sets to be enabled while others remain disabled. Moreover, we also relax the general bound M used in timed systems to detect quiescence, and allow different bounds for different sets of outputs. By considering the tiocoM theory, an enriched theory for timed testing with repetitive quiescence, and allowing the partition of input sets and output sets, we introduce the mtiocoM relation. A test derivation procedure which is nondeterministic and parameterized is further developed, and shown to be sound and complete wrt mtiocoM

Test Derivation from Timed Automata

A real-time system is a discrete system whose state changes occur in real-numbered time [AH97]. For testing real-time systems, specification languages must be extended with constructs for expressing real-time constraints, the implementation relation must be generalized to consider the temporal dimension, and the data structures and algorithms used to generate tests must be revised to operate on a potentially infinite set of states

Work-in-progress Assume-guarantee reasoning with ioco

This paper presents a combination between the assume-guarantee paradigm and the testing relation ioco. The assume-guarantee paradigm is a ”divide and conquer” technique that decomposes the verification of a system into smaller tasks that involve the verification of its components. The principal aspect of assume-guarantee reasoning is to consider each component separately, while taking into account assumptions about the context of the component. The testing relation ioco is a formal conformance relation for model-based testing that works on labeled transition systems. Our main result shows that, with certain restrictions, assume-guarantee reasoning can be applied in the context of ioco. This enables testing ioco-conformance of a system by testing its components separately

A Semantic Framework for Test Coverage (Extended Version)

Since testing is inherently incomplete, test selection is of vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to test coverage. Our starting point is a weighted fault model, which assigns a weight to each potential error in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour.We present several algorithms that, given a certain minimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of weighted fault models as fault automata. They are based on existing and novel optimization\ud problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol

A Semantic Framework for Test Coverage

Since testing is inherently incomplete, test selection is of vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to test coverage. Our starting point is a weighted fault model, which assigns a weight to each potential error in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour.We present several algorithms that, given a certain minimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of weighted fault models as fault automata. They are based on existing and novel optimization\ud problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol

Theories for model-based testing: real-time and coverage

In the last years, increasingly complex systems are being put in charge of critical tasks. When these complex systems, are drive by sophisticated software, they need to attain a high degree of reliability. Unfortunately, developing correct systems is difficult, and in the past there have been several complex systems that went wrong because they lacked serious analysis of their potential behaviour. In this thesis, we study an effective way of obtaining confidence on the correctness of a system, known as testing. Testing is the systematic process of finding errors in a system by means of extensively experimenting with it. In order to successfully test a system, it is crucially needed to count with both effective test cases and feasible strategies to execute them. Fortunately, work in formal methods helps us achieving this task in a precise and rigorous manner. A particularly successful formal theory of testing is the ioco theory, devised by Tretmans to work on labelled input-output transition systems. The theory smoothly covers issues like nondeterminism and quiescence (that is, the notion representing the absence of outputs). The ioco testing theory is clean and precise, and is the basis used in successful testing tools, like the TORX tool and the TGV tool. In this thesis we extend the ioco testing theory in three important directions, as follows. Our first extension concerns the addition of real-time, which is crucial to the analysis of several systems (e.g., systems where actions are required to occur in a precise moment). New models and formalisms that take into account real-time are introduced. Furthermore, we develop a new testing relation between these real-time models, and a sound and exhaustive algorithm to derive tests for that relation

Diagnosis and testing: How is their relation? Can they be combined?

Diagnosis and testing have coexisted for a long time, even though they have not been combined, mostly because they consider errors in a different manner. In this paper we present a novel framework that combines fault diagnosis with ioco-passive testing. To do so in a proper manner we initially present a formal definition of testability for transition system models, as well as for model-based testing. Later, we enrich our framework so that it captures possible attacks from malicious users. Finally, we consider a weighted failure model that can inform about the severity of a failure. We conclude that diagnosis and testing can be combined in a profitable manner.Fil: Brandan Briones, Laura. Universidad Nacional de Córdoba. Facultad de Matemática, Astronomía y Física; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba; ArgentinaFil: Madalinski, Agnes. Otto-von-Guericke-Universität Magdeburg; Alemani

Parallel Diagnosability Analysis with LTL-X Model Checking based on Petri Net Unfoldings

We present a framework that shows how components in parallel can infer the diagnosability property of the complete system (distributed and with multiple faults) from the diagnosability verification of each component synchronizing with a fault free versions of the other ones. Furthermore, we use existing efficient methods and tools, in particular parallel model checking based on Petri net unfoldings, to verifier diagnosability of such components